Short-Answer Responses to STEM Questions: Measuring Response Validity and Its Impact on Learning

Many studies in cognitive psychology have shown stronger learning benefits from short-answer questions than multiplechoice questions in student practices. However, multiple-choice is still, by far, the most popular form of assessment questions due to ease of grading. A hybrid method, in which students must first provide a short-answer response and then select from a list of multiplechoice options, offers a nice compromise that would allow students the learning benefit of short-answer responses while still keeping the grading simple and scalable. However, students often bypass the short-answer by entering in invalid (off-topic) responses to more quickly access the final multiple-choice selection, thereby bypassing any learning benefits offered by short-answer responses. In this work, we develop a novel method, GarbageSniffer, to automatically analyze and classify short-answer responses to questions as being valid (ontopic) or invalid (off-topic). We show that our method achieves excellent classification accuracy using real-world student data. We then analyze the impact of entering valid short-answer responses on future learning outcome using a separate real-world educational dataset taken from several high school AP Biology and Physics classes. In particular, we present evidence that students who make an effort to provide valid short-answer responses derive an educational benefit on later practice that is not seen in students who enter invalid short-answer responses. Keywords—Natural language processing, Machine learning, Cognitive psychology, Best educational practices.